Papers with low-memory scenarios
SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are an integral enabler of enterprise applications such as summarization, retrieval augmented generation, and agentic workflows. |
| Approach: | They propose a model transformation and distillation procedure that prefills later layers’ KV cache using an earlier layer’s output, allowing prompt tokens to skip those later layers. |
| Outcome: | The proposed procedure can reduce prefill computation by 25-50% across several LLM families while incurring minimum quality degradation. |